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ABSTRACT 
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^ Abstract . 

The intent of the study was to determine tli.. extent to which test 
statistics computed by the unweighted means analysis are F-distributed. 
Applicabili(y criteria were sought in terms of the number of factor 
levels and the degree to which cell frequencies differe. The unweighted 
means analysis, a frequently used approximate analysis, was contrasted 
with three least squares solutions. Evidence was relatively strong in 
favor of a least squares analysis if one is to conduct a two-factor 
analysis of variance for fixed effects. However, results confirmed that 
the approximate solution can be used with some confidence on main 
effects but. not interactions, when cell frequencies do not differ by more 
than four to one and factors exist at no more than four levels. 
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According wO Applebaum and Cramer (1974) the **nonorthogonal 
multifactor analysis of variance is perhaps the most misunderstood analytic 
technique available to the behavioral scientist, save factor analysis/' There 
is good reason to believe that such an assertion is substantially accurate 
even without the qualifiers. 

There appear to be multiple causes for the misunderstandings that 
exist. Recent work by Carlson and Timm (1974), Joe (1971), Rawlings 
(1972), and Ward and Jennings (1973) lead to the inference that a great 
deal of the confusion can be traced to texts that attempt to put in the 
hands of the user a set of convenient computational algorithms. The net 
effect of this practice has been to encourage practitioners to name their 
answers rather than specify their hypotheses. Thus two different individ- 
uals given the same set of data, both claiming to have performed **an 
unequal N*s analysis of variance," iiiay very well produce source tables 
with identical names for the answers but different numerical results. The 
obvious inference is that at least one of the individuals (possibly both) 
did something "wrong." In practice both may have conducted statistically 
defensible analyses for different hypotheses-neither explicitly stated. 

Complicating a consideration of the issue is the existence of what 
Applebaum and Cramer call "antiquated 'approximate' methods." One 
such approximate method is usually called an unweighted means analysis 
and many currently popular texts such as Dayton (1970), Glass and 
Stanley (1970), Kirk (1968), and Winer (1971), cover the topic in some 
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detail. By careful reading in these sources, one may infer that the analysis 
is approximate but none of them specify clearly what it is supposed to 
approximate. 

The origin of the method of "unweighted means analysis" can be 
traced to Yates (1934) who describes it as an approximate solution use- 
ful only when the class numbers do not differ very greatly. The purpose 
of this paper is to describe a simulation study investigating the properties 
of the unweighted means analysis and to contrast the results obtained 
with other methods based on least squares. 

Description of the Simulation 

Computer programs were used to sample repeatedly from populations 
with known distributions and to compute test-statistics for two factor 
designs. Resulting sampling distributions were compared with theoretical 
F-distributions in terms of expected values and the frequency of Type 
I errors. Those dimensions which were varied ?re given in Table 1. 
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Definitive sources 


for the procedures 


employed for computing test statis- 


tics can be found 


in Table 2. 






Insert Table 


2 about here 



ERIC 



5 



4 



Data Generation 

Random samples were drawn from normally distributed populations 
\vith homogeneous variance (CJ^^ = 1) about population means which 
ranged from 5 to 30. Each combination of the levels of variation for 
experimental variables was replicated one hundred times, requiring a total 
of 4 X 6 X 4 X 100 = 9,600 sets of data. A cell frequency pattern was 
generated for each of these combinations and maintained throughout /the 
one hundred replications. 

Population means were fixed according to the patterns specified in 
Table 1; e.g., the/^ij for pattern 1, 4 x 3 design were as follows: 



An 


= 5 


An- 


10 


A3- 


15 


An 


= 15 


A22 = 


10 


As- 


5 


An 


= 10 


An- 


15 


An- 


5 


A. 


= 10 




5 


A3 = 


15 



Each set of population means remained fixed for four hundred replica- 
tions-one hundred for each of the minimum/maximum cell frequency 
conditions. 

Data Analysis 

The distributions of test statistics which resulted from all four of the 
procedures given in Table 2 were compared with theoretical F-distributions 
in terms of Type I error .10,^= .05, and cxl= .01) and expected 

values. When the null hypothesis was true and both full and restricted 
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models were true, then all assumptions for F were met; thus test sta- 
tistics computed by all four methods, if accurate, should have been F- 
distributed. 

Main effects test resulting from fitting constants and unweighted 
regression when the full model was not true, i.e., interaction was present, 
were also compared with theoretical F-distributions even though an F- 
distribution was not expected. Conceivably, a researcher- could incorrectly 
assume no interaction. 

Computer Programs 

Even though general purpose computer programs for computing test 
statistics were available, new programs were written to increase efficiency. 
Thus cost for computer time was kept at a minimum. 

Accuracy of the computer programs was tested by comparing out- 
put with that of AVAR23 (Veldman 1967) and LINEAR (Ward & 
Jennings, 1973). AVAR23 and LINEAR are widely used to conduct 
unweighted means analysis and weighted squares of means analysis, 
respectively. Corresponding outputs were identical. Further pattern 4 
(see Table 1) was used to make a base line run with equal cell fre- 
qurencies. As anticipated, unweighted means, weighted squares of means, 
and fitting constants analyses produced identical values which were similar 
to the results from the unweighted regression analysis. 

Random Number Generation 

Function RANF (Laurens, 1970) was used to produce pseudo- 
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random numbers which were subsequently used in an algorithm developed 
by Ralston and Wilf (1967) to obtain a random point from an N(0, 1) 
distribution. Population means,^ jj were added to produce random points 
from jj 1) distributions. 

Summary of Results 
Results from the simulation runs were examined for discrepancies 
between (l).the number of observed Type I errors vs, the number 
expected and (2) the observed mean of the calculated F's vs. the 
expected value. In general the observed means of the calculated F's 
did not differ, .significantly from the expected values for any of the 
methods. Table 3 contains evidence of the extent to which the observed 
frequency of Type I errors differed from the expected frequency. 



Insert Table 3 about here 



A marked tendency existed for the unweighted means analysis to pro- 
duce more Type I errors than expected and for the least squares methods 
to produce fewer than expected. For example, the unweighted means 
interaction test at the .01 level produced almost twice (1.9028) as many 
errors as expected whereas the fitting constants method at the .01 level 
produced only 81% as many errors as expected. 

Summarized in Table 4 are a series of Chi Square goodness of fit 
tests, comparing observed and expected frequencies at the .10, .05 and 
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.01 levels. 



Insert Table 4 about here 



The most striking results in Table 4 i.s the lack of fit for the un- 
weighted means interaction test. Few trends are discernible although 
it should be noted that the weighted squares of means produced no 
significant chi squares. 

Even though both fitting constants and unweighted regression are 
inappropriate analyses in the presence of interaction, main effects tests 
were conducted when interaction existed in order to determine the 
consequences of using these analytic procedures inadvertantly. Thefie 
results are not reflected in Tables 3 and 4 but neither method resulted 
in valid tests. 
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Summary 

Summarizing, if one is to conduct a two-factor analysis of variance 
for fixed effects with unbalanced data, the evidence was relatively strong 
in favor of a weighted squares of means analysis, especially if a tCvSt for 
interaction is to be conducted. However, the approximate solution, U, 
can be used with some confidence to test for main effects when cell 
frequencies do not differ by more than four to one and factors exist at 
no more than four levels. Fitting constants and unweighted regression 
should not be used to conduct main effects tests unless the researcher 
is confident that interaction is negligible. 
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Footnote 

^For a definition of a true model, see Ward and Jennings, 1973, 
pp. 108-109 
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Table 1 

Summary of Experimental Variables 



V <inaDlc 




Levels of Variation 






Minimum 


Maximum 


1. Degree of inequality 


1. 




ID 


Oi Ceil Erequencies 






1 nn 






1 


1 

10 




A 

4. 


1 /C 
10 


zdO 


- - . - 

2. Number of levels in 


2, 


3, 4, 5, 


8, and 10 levels 


Factor A. 








(B Factor: held constant 








at 3 levels) 








3. Patterns of population 




A 


B A X B 


means (effects present) 


1. 


No 


No Yes 




2. 


Yes 


Yes No 




3. 


No 


Yes No 




4. 


No 


No No 
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Table 2 

Summary of Procedures Used for Computing Test Statistics 



Estimation Procedures 



1. Unweighted Means 



Weighted Squares of 
Means 



Fitting Constants 
(Main Effects only) 



4. Unweighted Regression 
(Main Effects only) 



Definitive Source 

Winer, 1971, pp. 402-404 ; 445-449 
Yates, 1934 



Carlson and Timm, 1974, pp. 564-565 
Yates, 1934 

Carlson and Timm, 1974, pp. 565-566 
Winer, 1971, pp. 404-414; 498-502 
Yates, 1934 

Kirk, 1968, pp. 204-208 



Carlson and Timm, 1974, p. 567: 

Fa = tSSg(y^^ ) - SSg(y/^o6./^)]/dfj 



SS,(^^^)/df2 
Fr = [SSg( ^^^) - SS 



/dfi 



SSJ 



Graybill, 1961, pp. 287-304 
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Unweighted 



Weighted 



Table 3 

Ratio of Number of Observed 
Type I Errors to Number Expected 



^ MAIN EFFECTS 
.01 ^ 1.3472 
.05 [ 1.1094 

i 

.10 ' .9806 



Fitting Constants 



Unweighted 
Regression 



.01 
.05 
.10 



.01 
.05 

.10 ' 

.01 [ 

.05 ' 

I 

.10 



.8472 
.9389 
.9264 

.8125 
.9208 

.9145 

.8125 
.9083 
.9167 



INTERACTION 
1.9028 
1.3111 
1.1486 

.9167 
.9500 
.9972 
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